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GEOGRAPHICALLY WATERMARKED IMAGERY AND METHODS 



Related Application Data 
This application is a continuation-in-part of application 09/800,093, filed March 
5, 2001, the disclosure of which is incorporated by reference. 

Field of the Invention 
The present invention relates to digital watermarking, and more particularly 
relates to digital watermarking as applied in the geophysical sciences. 

Background and Summary of the Invention 
Imagery commonly has a geographical aspect, e.g., a photograph may depict a 
particular place. It is often important to be able to precisely geographically locate a point 
within a picture or other image, e.g., for purposes of natural resources extraction, 
mapping, etc. 

In accordance with one embodiment of the invention, auxiliary data is 
steganographically embedded within an image to permit locations of points within the 
image to be determined. 

This and other features and advantages of the present invention will be more 
readily apparent from the following detailed description, which proceeds with reference 
to the accompany drawings. 

Brief Description of the Drawings 
Fig. 1 shows a (blank) image on which different grids and indicia relating to one 
embodiment of the invention are shown. 

Detailed Description 
Fig. 1 shows a representative (but blank) image 10 (composed of rows and 
columns of pixels, not particularly shown). For purposes of discussion, image 10 may be 
an aerial photo of land, but it should be recognized that imagery 10 is not so limited. 
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Due to various factors (e.g., camera lens artifacts, and the perspective from which 
the photo was taken, etc.), the rectangular photo typically does not depict a rectangular 
area of land. Instead, the area of land depicted may actually be trapezoidal, or of other 
shape. 

5 Overlaid on the Fig. 1 image are sample latitude and longitude lines 12, 14. 

These are virtual and do not appear in the actual image. (The straightness of the lines is 
unusual. In most landscape images, the receding horizon tends to curve any latitude or 
longitude projections that are not parallel to the image boundaries.) 

Each point depicted in the image 10 has a unique position that may be expressed 
10 by latitude and longitude (and, if 3D accuracy is desired, elevation) coordinates. In 
13 accordance with an illustrative embodiment of the invention, such position data for a 

single location depicted in the image is determined. (Various techniques can be 

JjW employed, e.g., reference to a pre-existing map or database, ground-truth measurements 

"H 

,p using GPS equipment, etc.) This location, and the pixel 16 corresponding thereto in the 

15 image, are termed the "arbitrary origin" in the discussion that follows. (For expository 
f convenience, the arbitrary origin in this discussion is the upper-left-most pixel in the 

{*& image, and the ground point corresponding thereto.) 

Z The image 10 is digitally watermarked across its extent with a payload that 

p| includes the coordinates of the arbitrary origin (e.g., latitude/longitude/elevation). In 
20 addition, the watermark payload also includes a parameter (e.g., angle 18) identifying the 
orientation of a vector pointing from the arbitrary origin to a known direction (e.g., true 
north). The watermark payload can also include a scale datum, e.g., indicating that 100 
pixels to the right (along the row) from the arbitrary origin corresponds to a distance - on 
land - of 250 yards. 

25 As noted, a rectangular image generally does not depict a rectangular piece of 

land. Moreover, even if a photo is taken from directly overhead - using a lens that 
introduces no aberrations - there is the slight complication posed by the fact that 
longitudinal lines are not parallel, but meet at the poles. Accordingly, if high accuracy is 
desired, the watermark can additionally convey coefficients for one or more polynomials 

30 (e.g., one for each coordinate axis), which model the apparent warp of the photographic 
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depiction along different axes. (In an exemplary arrangement 5 coefficients of 8 bits 
each are provided for the latitude and longitude polynomials, and 6 coefficients of 8 bits 
each are provided for the elevation polynomial.) 

Thus, in an exemplary embodiment, the watermark payload may comprise the 
5 following (196 bits total) : 



Arbitrary origin latitude 


20 bits 


Arbitrary origin longitude 


20 bits 


Arbitrary origin elevation 


16 bits 


Orientation vector 


12 bits 


Polynomial coefficients that model warp in latitude dimension 


40 bits 


Polynomial coefficients that model warp in longitude dimension 


40 bits 


Polynomial coefficients that model warp in elevation dimension 


48 bits 



Digital watermarking is not belabored in this specification because such 
technology is well understood by artisans in the field of steganography. Briefly, 

1 0 however, watermarking typically works by making subtle changes to the brightness of 
image pixels, conveying message payloads that can be detected by suitable detector 
software or hardware. The embedding process generally adjusts to the unique 
characteristics of the image, placing a stronger watermark signal in areas with rich detail 
and a weaker watermark signal in areas with little detail. Because the payload is carried 

15 by the image's pixels, it is file-format independent. The payload can survive most normal 
processing operations, such as compression, edits, file format transformations, copying, 
scanning and printing. Some watermarking techniques are also robust against rotation 
and scaling, e.g., through use of embedded calibration data, or auto-correlation 
techniques. 

20 Any watermarking technique can be employed in the present invention, provided 

the requisite number of watermark payload bits can be embedded without introducing 
objectionable corruption into the image. Suitable watermarking technologies are 
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disclosed, e.g., in patent publications US 6,122,403 US 6,044,182, and WO 99/45705, 
and in pending US application 09/503,881 . 

In a particular embodiment, the watermark payload is represented in a single 128 
x 128 pixel patch 20, which is then tiled across the image (with local scaling to reduce 
visibility). Each patch comprises 16,384 pixels. In Fig. 1, one of the patches 20 is 
indicated by cross-hatching for ease of identification. 

To enhance robustness, the watermark payload may be processed, e.g., by BCH, 
Reed-Solomon, convolutional, or turbo coding, or the like, to provide error 
detecting/correcting capability. Such coding has the effect of transforming the 196 bit 
payload bits into, e.g., 320 bits ("raw bits"). Each of the 16,384 pixels in the patch is 
encoded with one of these raw bits, so that each such bit is represented about 50 times per 
patch. The pixels corresponding to a single raw bit are desirably distributed across the 
patch, so that severe corruption of a small area of the watermarked image does not 
irretrievably lose certain raw bits. 

On the detection side, the image is processed to retrieve the 320 raw bits, and then 
the 196 payload bits are determined from the raw bits. From these payload bits, a user of 
the image knows the geographical coordinates of the point at the arbitrary origin and, 
through use of the other encoded parameters, can deduce the geographical location of any 
other point depicted in the image. 

In other embodiments, more elaborate watermark encoding can be used. For 
example, instead of tiling the identical watermark patch over and over across the image, 
each patch can be slightly different, e.g., encoding the position of that tile within the 
array of tiles. In one arrangement the tile position data is a pair of numbers indicating 
tile-row/tile-column offsets from the tile containing the arbitrary origin. Referring to Fig. 
1, the tile containing the arbitrary origin 16 may be designated {0,0} . The tile next to it 
in the row may be designated {0, 1 } , etc. These index values may be encoded as 5 bits 
each, which bits are included in the watermark payload. This arrangement offers 
advantages in environments in which image cropping, rotation, or other image 
transformations may occur. By decoding the payload from a watermark tile, its location 
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relative to the arbitrary origin can be determined, and the location of the arbitrary origin 
16 can thus be inferred (even if that point has been cropped out of the image). 

In still other embodiments, information about the image perspective can be 
conveyed through a watermark. Various forms of representation are possible. In one, the 
image perspective data can comprise the compass angle at which the camera is pointing 
(6), and the elevation angle between the arbitrary origin point and the camera (0). The 
former may be represented, e.g., by 10 bits, the latter by 8. Additionally or alternatively, 
the perspective data can identify the lens or its attributes, so that optical distortion of the 
image can be characterized. In an index-based system, a six-bit code can be used to 
identify one of 64 different lenses. 

With different types of imaging systems, different forms of perspective 
information may be appropriate. For example, in so-called "whisk broom" cameras (i.e., 
those that repeatedly acquire line scans from a moving viewpoint), the perspective 
information may additionally include the starting and ending positions (the latter may be 
expressed as an offset from the former, allowing some payload conservation). 

In yet other embodiments, elevation data for different points in the image can be 
encoded through watermarks. In one such arrangement, elevation data is determined for 
points at 64-pixel gridded spacings across the image. These points are designated in Fig. 
1 by the stars labeled 22. (Only a few such stars are shown in Fig. 1. The arbitrary origin 
16 is also such a point.) The elevation may be expressed in absolute terms (e.g., feet 
above sea level), or relative to another reference (e.g., the elevation of the arbitrary 
origin). Again, 16 bits per elevation may be used. (Or if difference in elevation from the 
arbitrary origin is used, then 8-12 bits may suffice.) 

In one such arrangement, the elevation data for each starred point is watermark- 
encoded in a 64 pixel by 64 pixel subpatch 24 centered around the star. Again, one such 
sub-patch 24 is shown in Fig. 1 by cross-hatching for ease of identification. More 
generally, these sub-patches 24 are the regions bounded by the fine, dotted lines in Fig. 1. 
Again, sub-patches 24 are tiled across the image, but each one conveys a (typically) 
different elevation payload. 
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(In the Fig. 1 arrangement, each patch 20 encompasses one full sub-patch 24, and 
parts of eight others. By this arrangement, elevation data is encoded for the points at 
each corner of each patch 20, as well as for the point at the center of the patch, and at 
points mid-way along each patch side boundary. In other embodiments, of course, sub- 
5 patches 24 can be sized and positioned differently relative to patches 20.) 

The elevation watermark, based on patches 24, maybe simply overlaid on the 
main watermark, based on patches 20. Desirably, however, there is some coordination 
between the two watermarks, so as to avoid extreme changes in any pixel values (as may 
occur, e.g., if both watermarks try to change a pixel by a maximum amount permitted by 
„ 10 the respective watermarking technique). In one such coordination arrangement, each 
=11 pixel in the image is assigned to one of the two watermarks. For example, 50-80% of the 

Jj pixels in the image may be assigned to the main watermark, and 50-20 % may be 

assigned to the elevation watermark. The assignment may be done based on a regular 
p array, or a stochastic assignment may be used. In some cases, it may be prudent to 

15 allocate extra pixels to carry the elevation payload where - as in the upper right - the 
f># sub-patch 24 extends beyond the boundary of the image, limiting the number of pixels to 

ffj convey elevation data (e.g., for arbitrary origin point 1 6). 

C6 

J5g (This coordination technique has applicability beyond the present context, and is 

|# generally applicable to a variety of watermarking applications in which plural 
20 watermarks are used.) 

By techniques such as the foregoing, an image can be provided with extensive 
photogrammetric information that travels with the image, notwithstanding distortion, 
cropping, format conversion, etc. 

This data can be exploited in various ways. One utilizes a computer system on 
25 which the image is displayed, e.g., on a monitor or screen. An operator uses an input 
device, such as a mouse, light pen, graphics tablet, or the like, to designate a particular 
point in the displayed image. In response to selection of the point (by clicking or other 
known selection technique), the computer processes the embedded watermark 
information and displays to the operator the precise latitude, longitude and elevation of 
30 the selected point. 
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Using the elevation data, the computer system can also generate, and render, a 3D 
view of the depicted landscape, from an arbitrary viewing angle. Surfaces that are hidden 
in the original image may be extrapolated using known techniques, and presented in a 
different color or texture to indicate their synthetic basis. 
5 In embodiments in which the camera perspective is known, the projections of 

latitude and longitude lines on the depicted terrain can be adjusted, e.g., in accordance 
with variations in elevation. If the camera perspective is such that it is viewing down a 
downwardly-inclined slope, for example, the latitude or longitude lines that traverse this 
slope can be virtually placed more closely spaced together than would be the case if the 
10 camera view were orthogonal to the slope, 
rjj The mathematical manipulations associated with such operation are somewhat 

;jf complex, but well within the skills of those working in the photogrammetric and mapping 
S arts. To determine elevation at an arbitrary point and to generate 3D models, for 

Q example, the elevations at the starred points 22 are provided to an algorithm that applies a 

^ 15 bi-cubic spline-fitting model so as to estimate the elevation at any point on the image. 
i»* (The computer system can take various forms, but most include conventional 

fjj computer components such as one or more CPUs, volatile storage (e.g., RAM), non- 
w volatile storage (e.g., ROM, fixed and removable magnetic disks, fixed and removable 
U optical disks), interfaces (e.g., WAN, LAN, USB, modem, serial), input/output devices 
20 (e.g., monitor, keyboard, mouse, tablet, joystick, light pen), etc. Associated with the 
computer system is various software, including operating system software and 
applications software - the latter being programmed to perform the data processing and 
presentation operations detailed above. Naturally, such programming can be stored on 
fixed or removable computer storage media. In some embodiments, parallel or 
25 distributed computer architectures may be employed, e.g., with different components of 
the computer system being located remote from each other.) 

A variety of aerial mapping and associated image database techniques can be used 
in conjunction with the present invention. Representative systems are shown, e.g., in 
patents 5,608,405, 5,926,581, 5,974,423, 6,023,278, 6,177,943, 5,995,681, 5,550,937 and 
30 6,150,972. 
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To provide a comprehensive disclosure without unduly lengthening this 
specification, applicant incorporates by reference the patents publications and 
applications cited above. 

Having described and illustrated the principles of the invention with reference to 
illustrative embodiments, it will be recognized that the invention can be modified in 
arrangement and detail without departing from such principles. 

For example, while the "arbitrary origin" was the pixel in the upper-left corner of 
the image in the illustrative example, this placement is not critical. The arbitrary origin 
can be moved to any location, with relative measurements being adjusted accordingly. 

Likewise, while the detailed embodiment contemplated that the coordinates of the 
arbitrary origin are literally encoded as part of the digital watermark payload, in other 
embodiments this need not be the case. Instead, e.g., the watermark payload can be an 
arbitrary identifier that identifies an entry in a data structure (e.g., table or database) in 
which the coordinate data is stored. The same index-a-remote-store approach can be used 
with any of the other payload data. 

Although latitude/longitude/elevation were used as exemplary coordinates, it will 
be recognized that other coordinate geometries can alternatively be employed. 

The main watermark payload is described as including coordinate, orientation, 
scale, and polynomial correction data. Depending on the application, certain of this data 
may be omitted, and/or certain additional information may be included in the watermark 
payload. The payload length is exemplary. Some embodiments can employ a payload 
that is considerably shorter (e.g., by abbreviating the bits dedicated to each data and/or 
omitting certain data). Other embodiments may employ a payload that is longer. 

While elevation may be expressed in height above sea level, this need not be the 
case. Height relative to any other measure can alternatively be employed. 

The illustrative embodiments' encoding of plural data (e.g., coordinate data, and 
lens data) in a single watermark payload is not essential. In other embodiments, the 
different elements of embedded information can be conveyed through distinct 
watermarks, e.g., layered over each other, interspersed between each other, coordinated 
with each other in the manner of the elevation watermark, etc. 
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In many embodiments, lossless data compression techniques (e.g., Lempel-Ziv 
based) can be employed to reduce the number of payload bits that are encoded in a 
watermark. 

Although described in the context of watermarking for mapping and 
photogrammetric purposes, the principles detailed herein find application in many other 
watermarking applications, not limited to the purposes particularly detailed. 

In view of the many embodiments to which the principles of the invention may be 
applied, it should be recognized that the detailed embodiments are illustrative only and 
should not be taken as limiting the scope of the invention. Rather, I claim as my 
invention all such embodiments as fall within the scope and spirit of the following 
claims, and equivalents thereto. 



